fix(core): long cut-list edits play every cut without dropped or late frames by miguel-heygen · Pull Request #6785 · heygen-com/hyperframes

miguel-heygen · 2026-10-30T03:30:25Z

What

Three fixes for Studio preview on a long edit cut from one source file:

At a cut, the outgoing video plays its last frame before the out-point instead of stopping about a frame early, and no frame at or past the out-point shows, even when the decoder drops the frame before it.
A composition with no animation timeline no longer rewrites every hidden clip twice on each frame of playback.
A short clip that has already downloaded in full stays loaded when the playhead moves away, so it plays on time when its turn comes. Before, a b-roll clip could show its first frame over a second late.

Why

fix(core): a video no longer runs ahead of the playhead or past its out-point #6438 pauses a playing video once its currentTime is within one sync tolerance (40 ms) of its out-point. On 23.976 fps footage a frame lasts 41.7 ms, so whenever the last frame before the out-point starts inside those 40 ms, the video is paused before that frame is presented and the picture holds the frame before it until the cut.
On each forward seek the runtime briefly shows the hidden clips whose tweens draw for the first time, so GSAP reads their authored style, then hides them again. With no captured timeline firstRendered is null, null?.size !== 0 is true, and it took the fallback that shows every hidden clip: on a 140-clip edit, about 33,000 attribute writes a second, each a style invalidation on a video.
The preview releases a clip that leaves its look-ahead window (drops its src and reloads it) so it stops holding one of the browser's six connections. A clip armed by the clip before it loads early; when that clip ends the arm lapses and the clip is released even when its whole file had arrived and it held no connection. It then fetched again just before its start, queued behind long video downloads on the same host, for 1.2-3.2 s.

Related work

Refines #6438 (the out-point hold). #6033 bounds open-ended media ranges in the file servers, which shortens how long long downloads hold connections; fix 3 is independent of it. The reveal check in fix 2 was last changed by #6189 and #5690. Folds in #6786 and #6787.

How

Where the browser has requestVideoFrameCallback, a playing video within 0.5 s of its out-point is watched frame by frame. When a presented frame plus the longest frame step seen in this watch would land on or past the out-point, the video is paused there. The tick check holds a watched video once it reaches the predicted last frame before the out-point (on the shortest step seen), so a dropped frame cannot carry it onto the out-point's own frame; a paused video shows the frame at its currentTime, so that last frame still shows. A seek between two frames is not taken as a frame step. The 40 ms rule stays where the browser cannot report frames. The hold is dropped wherever the existing lead hold is dropped.
The reveal also needs a timeline that can draw: the root timeline or a registered scene timeline. The "registered timelines other than the root's" list moves into one helper that the scene seek and this check both use.
releaseMedia returns early when the element's buffered range covers the whole file. Its only caller is the look-ahead window in init.ts.

Test plan

Unit tests: media.test.ts (a video inside the old 40 ms margin plays to its last frame and holds; a seek back from the hold plays again; a dropped last frame still holds before the out-point's frame; an uneven cadence holds before a long step crosses the out-point), init.test.ts (seeks on a page with no timeline never query [data-hf-clip-hidden]), preloadMedia.test.ts (a whole-file clip is kept; clips with 0-15 s or 430-2642 s of a long file buffered are released).
fails without the fix:
- restoring the 40 ms margin, or dropping the out-point hold from the hold check, fails plays a video in step with the playhead up to the last frame before its out-point; dropping the reset on a seek fails plays a held video again once a seek takes the playhead back; holding at the out-point instead of the predicted last frame fails holds a video on its last frame before the out-point when that frame is dropped; predicting from the last step instead of the longest fails holds a video with an uneven cadence before a long step crosses the out-point.
- removing the timeline condition fails leaves hidden clips alone on every jump when the page has no timeline.
- removing the early return fails keeps a clip whose whole file is buffered; treating any range that reaches the end as whole fails releases a clip with only 430-2642 s of a long file buffered; treating any range from the start as whole fails releases a clip with only 0-15 s of a long file buffered.
- Each mutation fails only that test.
vitest run src/runtime in packages/core: 108 files, 2653 tests passed. oxfmt, oxlint, core tsc and the comment ratchet clean.
Manual testing in Studio preview, Chrome 156 on an M4 Pro MacBook Pro (macOS 26.5, 120 Hz display), playback started with Studio's Play button, main against each fix, 3 runs per side. Media is one synthetic 44-minute 1080p H.264 file at 23.976 fps with long keyframe gaps (7 s apart on average, up to 10.4 s). The measured builds predate the dropped-frame guard in fix 1, which is covered by its unit test only.

A 149-clip edit of that file (graded clips, b-roll overlays), 60 s per run, picture timed from each video's requestVideoFrameCallback:

per run, `main` → all three fixes	0-60 s	64-124 s
dropped frames (a frame held 2+ frame periods)	5, 6, 3 → 1, 1, 1	8, 6, 6 → 3, 2, 2
cuts whose last frame never showed	5, 3, 5 of 17 → 0 of 17, 0 of 16, 0 of 17	3, 2, 2 of 12 → 0 of 12, 0 of 11, 0 of 12
frames shown past an out-point	0 → 0	0 → 0
b-roll clips whose first frame came over 100 ms late	1, 1, 1 (1.20-1.27 s) → 0, 0, 0	0 → 0
frames over 25 ms on the main thread	1, 1, 2 → 0, 0, 0	1, 1, 1 → 0, 1, 1

Fix 1 alone on 64-124 s: longest picture hold 121, 108, 99 → 84, 83, 84 ms. Fix 3 alone on 0-60 s: latest first frame after the playhead reached a clip 1211, 1215, 1280 → 33, 29, 26 ms.

The same 140 clips with no GSAP timeline, 30 s runs, fix 2 alone:

per run, `main` → fix 2
`data-hf-clip-hidden` writes a second	33,340, 33,357, 33,365 → 1, 1, 1
main thread idle in a 20.5 s CPU profile	6.86 s → 8.66 s (about 88 ms a second freed)
frame interval p95	10.2-10.4 ms → 10.3-10.4 ms

On this machine the time fix 2 frees does not show up as smoother frames; the work goes away.

For scale: one plain <video> playing the same file on the same machine drops 1-2 frames a minute.

Documentation updated (not applicable)
Comments follow CONTRIBUTING.md "Comments"

github-actions · 2026-10-30T03:48:33Z

Edit accuracy: accurate 271 (base branch 271), smooth 127 of those

The gate passes.
Smoothness is reported in the artifact, not gated. A case fails only if it fails 2 of 3 runs.

Quarantined, measured but not gated (0)

Unstable (1)

seqplainkeys-none-center-r0-root-z100: tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo false, teleport true / tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo true, teleport true / tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo true, teleport true

heygengenesis

Verdict: APPROVE · 0 blocker · 1 important · 0 nits · c3a36ff2

Both new tests fail with main's media.ts, as the fails without the fix: line says, and the 176 tests in media.test.ts pass at this head. CI is green; the PR is blocked only on review. The one finding, posted inline, doesn't block: it can expose at most about one frame, and only when the browser drops the last frame before the cut.

Scope

Read all of media.ts and media.test.ts. Checked every place the hold is cleared: hard, strict and force sync, evictMediaSyncState from both mediaProxy evict sites and the inactive branch. A transport pause while held keeps the out-point hold, and resuming stays paused, which is correct. A seek back over 0.5 s clears it (offsetJumped). Probed a dropped-frame case locally; details are in the inline comment.

…ad of stopping a frame early

…-point's frame show

…very clip each frame

… time

heygengenesis

Verdict: APPROVE — 1 resolved, 0 open, 0 new · c3a36ff2 → 1fa435d4. I checked the six fails-without mutations: each new test fails against the earlier media.ts or main's init.ts/preloadMedia.ts. CI is green.

…eygen-com#7271) 537b88d ported heygen-com#7271, whose muteCallbacks list spreads sceneTimelines(), but that helper arrives upstream with heygen-com#6785, which this fork has not taken. The default tsconfig excludes the runtime, so only CI's `tsc -p tsconfig.runtime.json` caught the missing name (TS2552 at init.ts:3982). Add upstream's helper as is: registered timelines other than the root composition's. Co-authored-by: Vance Ingalls <vance@heygen.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The 1a9bb89 port of heygen-com#7402 and heygen-com#7378 brought in releaseMedia(el, flag) with window.__HF_RELEASE_PASSED_MEDIA (heygen-com#6929, needs the heygen-com#6785 buffered guard) and registerSeekCompletion(landed, "image-decode") (the SeekHold union from heygen-com#6596). Both upstream commits stay deferred, so typecheck:runtime failed in the Windows render verification workflow. The fork's releaseMedia already frees every clip, so the call keeps its behaviour. Add tsconfig.runtime.json to the pre-commit typecheck, which only covered tsconfig.json (it excludes src/runtime), and pin it in the hook test. Co-Authored-By: Claude Sonnet <noreply@anthropic.com>

heygengenesis Bot approved these changes Oct 30, 2026

View reviewed changes

Comment thread packages/core/src/runtime/media.ts Resolved

miguel-heygen added 6 commits October 30, 2026 00:28

fix(core): a video plays to the last frame before its out-point inste…

fbf69b9

…ad of stopping a frame early

fix(core): a dropped frame before an out-point no longer lets the out…

6e403a9

…-point's frame show

perf(core): a video edit with no animation timeline stops rewriting e…

2a44408

…very clip each frame

refactor(core): drop the comments the comment ratchet counts

c0bfb06

fix(core): a clip that has fully downloaded stays loaded and plays on…

331cbab

… time

refactor(core): drop the comments the comment ratchet counts

1fa435d

miguel-heygen force-pushed the fix/last-frame-before-out-point branch from c3a36ff to 1fa435d Compare October 30, 2026 04:33

miguel-heygen changed the title ~~fix(core): a video plays to the last frame before its out-point instead of stopping a frame early~~ fix(core): long cut-list edits play every cut without dropped or late frames Oct 30, 2026

This was referenced Oct 30, 2026

perf(core): a video edit with no animation timeline stops rewriting every clip each frame #6786

Closed

fix(core): a clip that has fully downloaded stays loaded and plays on time #6787

Closed

heygengenesis Bot approved these changes Oct 30, 2026

View reviewed changes

miguel-heygen added this pull request to the merge queue Oct 30, 2026

Merged via the queue into main with commit 9fd90d2 Oct 30, 2026
132 of 133 checks passed

miguel-heygen deleted the fix/last-frame-before-out-point branch October 30, 2026 05:17

This was referenced Oct 30, 2026

fix(core): the first cut after Play starts on time instead of running ahead #6794

Merged

chore: release v0.8.221 #6802

Merged

Navigation Menu

Repository navigation

fix(core): long cut-list edits play every cut without dropped or late frames - #6785

fix(core): long cut-list edits play every cut without dropped or late frames#6785
miguel-heygen merged 6 commits into
mainfrom
fix/last-frame-before-out-point

miguel-heygen commented Oct 30, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 30, 2026 •

edited

Loading

Uh oh!

heygengenesis Bot left a comment

Uh oh!

Uh oh!

heygengenesis Bot left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

miguel-heygen commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

Related work

How

Test plan

Uh oh!

github-actions Bot commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Edit accuracy: accurate 271 (base branch 271), smooth 127 of those

Uh oh!

heygengenesis Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

heygengenesis Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

miguel-heygen commented Oct 30, 2026 •

edited

Loading

github-actions Bot commented Oct 30, 2026 •

edited

Loading